Psychometric validation for a brand-new tool for the assessment of executive functions using 360° technology

EXecutive-functions Innovative Tool 360° (EXIT 360°) is an original 360° instrument for an ecologically valid and multicomponent evaluation of executive functioning. This work aimed to test the diagnostic efficacy of EXIT 360° in distinguishing executive functioning between healthy controls (HC) and patients with Parkinson’s Disease (PwPD), a neurodegenerative disease in which executive dysfunction is the best-defined cognitive impairment in the early stage. 36 PwPD and 44 HC underwent a one-session evaluation that involved (1) neuropsychological evaluation of executive functionality using traditional paper-and-pencil tests, (2) EXIT 360° session and (3) usability assessment. Our findings revealed that PwPD made significantly more errors in completing EXIT 360° and took longer to conclude the test. A significant correlation appeared between neuropsychological tests and EXIT 360° scores, supporting a good convergent validity. Classification analysis indicated the potential of the EXIT 360° for distinguishing between PwPD and HC in terms of executive functioning. Moreover, indices from EXIT 360° showed higher diagnostic accuracy in predicting PD group membership compared to traditional neuropsychological tests. Interestingly, EXIT 360° performance was not affected by technological usability issues. Overall, this study offers evidence that EXIT 360° can be considered an ecological tool highly sensitive to detect subtle executive deficits in PwPD since the initial phases of the disease.

Virtual reality-based (VR) evaluation can be considered a new promising paradigm for neuropsychological assessment, able to provide an ecological evaluation of everyday cognitive impairments predicting real-world performance 1 , compared to traditional paper-and-pencil or computerized neuropsychological batteries 2 . In the last few years, advances in 360° technology emerged as an interesting alternative approach to create VR content by recording familiar environments before, and then showing them to participants on a head-mounted display (HMD) 3,4 . Furthermore, 360° settings allow participants to be evaluated in virtual scenarios that they experience from a first-person perspective, overcoming some technical (e.g., high skills or cost) and clinical (e.g., vertigo, nausea) limitations of VR 1,5 . Borgnis and colleagues (2021) have developed EXecutive-functions Innovative Tool 360° (EXIT 360°), an original 360° instrument for an ecologically valid and multicomponent evaluation of executive functioning 6 . Participants are engaged in a "game for health" in which they are immersed in 360° household environments delivered via a conventional HDM. In these scenarios, subjects have to complete seven everyday subtasks designed to assess several aspects of executive functioning simultaneously and quickly. The need to develop EXIT 360° arose from the literature that showed the inability of traditional neuropsychological tests to detect executive impairments in everyday situations 7,8 . However, executive dysfunction constitutes a significant public health challenge in several psychiatric and neurological pathologies due to their high impact on personal independence (e.g., preparing meals, managing money, shopping, using a telephone), ability to work, educational success and social relationships [8][9][10] . Consequently, its ecologically valid assessment must be early and adequate to plan timely rehabilitation interventions 11,12 . Over the years, several studies have shown the feasibility and www.nature.com/scientificreports/ acceptability of VR-based tools in the early assessment of executive functioning in healthy controls and many psychiatric and neurologic pathologies (see review 13 ). However, a recent systematic review (2022) has shown several psychometric issues in the available VR-based assessment tools for executive functioning due to limited studies on construct validity, discriminant validity, usability, and test re-test reliability 13 . To date, EXIT 360° appears to be an innovative and interesting solution within the field of neuropsychological evaluation of executive functionality since preliminary studies involving healthy control subjects showed excellent results in construct validity 14 and usability 15,16 . Firstly, EXIT 360° can be considered a usable, learn-to-use, clear, enjoyable, attractive, and friendly assessment tool, regardless of demographic characteristics (age and education) or technological expertise. Moreover, it appeared to be a fast, original, engaging, and challenging tool able to evaluate real-life impairments in several components of executive functioning (e.g., planning, decision-making, problem-solving, attention, visual searching, and working memory) with irrelevant adverse effects.
In light of these characteristics and psychometric proprieties, EXIT 360° could represent a promising tool for evaluating executive dysfunctions in several clinical populations, such as Parkinson's Disease (PD). Indeed, in addition to the well-documented motor symptoms (e.g., bradykinesia, resting tremor, rigidity), people with PD frequently develop a wide range of non-motor symptoms (NMS) since the initial stages of the disease course, even before the onset of motor symptoms in the prodromal state [17][18][19] . Cognitive dysfunction represents one of the major clinical NMS of PD 17 , and executive dysfunction is the best-defined cognitive impairment in early-stage non-demented PD 20 , particularly affecting attention, planning, deduction, inhibition, capacity to elaborate a strategy, set-shifting (flexibility) and working memory 21 . As a result, patients have trouble in many goal-directed everyday activities, with negative implications for daily functioning (i.e., preparing meals, managing money, shopping, and work) 12,22,23 and quality of life [23][24][25] . Interestingly, an increasing number of longitudinal studies suggested that early executive dysfunction is predictive of PD conversion in "PD with dementia" 26,27 . However, some studies showed that traditional standard assessment appeared not sensitive to the early detection of executive deficits 22 . Therefore, EXIT 360° could permit early detection of executive deficits and, consequently, identify patients at risk of developing dementia, providing early neurorehabilitative interventions 22,28 .
This study wants to deepen EXIT 360° diagnostic efficacy in distinguishing executive functionality between healthy subjects and patients with PD, also compared with traditional neuropsychological tests for executive functioning. It should be noted that the lack of discriminant validity constitutes a significant limitation in the use of a specific tool since the absence of information on diagnostic specificity and sensitivity in clinical populations makes it impossible to introduce them into clinical practice. The study was approved by the "Fondazione Don Carlo Gnocchi-Milan" Ethics Committee. The neuropsychologist provided all participants with a complete explanation of the purpose and risk of the study before they signed the written informed consent based on the revised Declaration of Helsinki (2013).

Materials and methods
All participants underwent a one-session evaluation at FDG that involved three main phases: (a) neuropsychological evaluation, (b) EXIT 360° session, and (c) usability assessment 32 .
Neuropsychological evaluation. All subjects performed, in a clinical setting and before EXIT 360° completion, a neuropsychological evaluation of global and executive functioning, conducted by a trained neuropsychologist using conventional pencil-paper tests: EXIT 360° session. Each subject underwent an evaluation with EXIT 360°, preceded by a familiarization phase with the device and virtual environment to control any adverse effects (e.g., dizziness, nausea). Detailed characteristics and administration procedure of EXIT 360° have been recently described in a published study protocol 32 . Briefly, EXIT 360° is a 360°-based assessment tool for a complete assessment of executive functioning, engaging participants in a "game for health" delivered via smartphones, in which they have to perform seven subtasks (e.g., Unlock the Door; Choose the Person; Turn on the light) in 360° domestic environments (e.g., liv- www.nature.com/scientificreports/ ing room and bedroom). All participants sit on a swivel chair and wear the mobile-powered headset that allows for immersing in virtual environments explorable through the head's movement as in real-life situations 42 . The test started when they heard, "You are about to enter a house. Your goal is to get out of this house in the shortest time possible. To exit, you will have to complete a path and a series of tasks that you will encounter along your way. Are you ready to start?". To respond to each task, subjects had to choose between three or more alternatives by moving their head and positioning a small white dot that they saw in the headset on the answer for a few seconds. Participants had to perform all seven subtasks, obtaining one point for a wrong answer or two for a correct one. Overall, EXIT 360° allowed for the collection of Total Score (range 7-14) and Total Reaction Time (i.e., time spent to solve each of the 7 tasks, excluding instructions time. Then, the Total Reaction Time of EXIT 360° was calculated by summing the single reaction times).
Usability assessment. All participants underwent a usability assessment of the technological tool using the System Usability Scale (SUS), a short questionnaire of 10 items on a 5-point scale from "completely disagree" to "strongly agree" 43,44 .
Statistical analysis. Descriptive statistics included the frequencies, percentages, median and interquartile range (IQR) for categorical variables and the mean and standard deviation (SD) for continuous ones. The normality of data distribution was assessed using the Kolmogorov-Smirnov test. Demographics and clinical characteristics of the two groups (i.e., PD and HC) were compared using a t-test for independent samples (parametric or non-according to variables) or a chi-squared test for categorical variables. Moreover, ANOVA one-way between subjects was conducted to assess any differences between groups in traditional neuropsychological tests and EXIT 360° scores. Pearson's correlation was applied to evaluate the possible relationship between the scores of the conventional neuropsychological tests and EXIT 360° scores (Total Score and Total Time). Finally, ROC curves evaluated all the tests' specificity and sensitivity. Regarding system usability, Pearson's correlation was conducted to compare EXIT 360° scores with the usability score. Moreover, ANOVA between subjects was performed to evaluate any differences in usability between the two groups. All statistical analyses were performed using Jamovi 1.6.7. Corrections for multiple comparisons were performed using the online calculator of False discovery rate correction for multiple comparisons. A p-value of < .05 was considered statistically significant. Nonlinear stochastic approximation (i.e., machine learning) methods were used to compare the classification accuracy of traditional neuropsychological assessments versus the EXIT 360° indices for classifying participants into either the "Patients with PD" or "Healthy Controls" groups. Different machine learning algorithms were employed: Logistic Regression and Support Vector Machine algorithms. All these analyses were computed using Python 3.4.
Finally, we conducted an additional analysis dividing the PD sample into two groups according to performance on traditional neuropsychological (NPS) tests for executive functioning: Group PD_NPS + (i.e., patients with pathological/borderline performance in at least one NPS test) and Group PD_NPS-(i.e., patients that reported deficits in everyday activities linked to executive functioning-e.g., managing money or cooking-with a normal performance at NPS). We performed ANOVA between groups (PD_NPS + ; PD_NPI-; HC) to show any difference between the three groups in EXIT 360° and NPS scores. The results were reported in supplementary materials.
Ethics approval. The study was conducted according to the guidelines of the Declaration of Helsinki and approved by "Fondazione Don Carlo Gnocchi-Milan" Ethics Committee.
Informed consent statement. Informed consent was obtained from all subjects involved in the study.

Results
Participants. Table 1   www.nature.com/scientificreports/ Traditional neuropsychological evaluation. EXIT 360°. Correlation between neuropsychological tests and EXIT 360°. Table 4 shows significant correlations (Pearson's correlation) between traditional paper and pencil neuropsychological tests and the two scores of EXIT 360°.
Classification of healthy controls or clinical group. The performance of the classifiers was evaluated by carrying out a relative operating characteristic (ROC) analysis. The area under the ROC curve (AUC) provides a single measure of overall prediction accuracy. Specifically, ROC curves investigated the diagnostic accuracy of EXIT 360° showing that (Fig. 1): • EXIT Total Score ≤ 11 could accurately discriminate HC and PwPD groups, with high sensitivity (90.91%) and specificity (77.78%) (AUC = 0.897-excellent accuracy value).   Further analyses showed EXIT Total Score ≤ 11 can discriminate between HC and PwPD with better overall prediction accuracy, sensitivity, and specificity than MR (DeLong test-p < .001) and FAB (DeLong test-p = .04) scores (Fig. 2).

PwPD mean (SD) HC mean (SD) Group comparison (p-value) Corrected p-value
Moreover, EXIT Total Time > 600 allows for discriminating between HC and PwPD with better overall prediction accuracy, sensitivity, and specificity than TMT-B and TMT-BA (DeLong test-p < .001) scores (Fig. 3).
In addition, nonlinear stochastic approximation methods confirmed the ROC analyses, showing an excellent accuracy of EXIT 360° scores in discriminating HC and PwPD. Results showed a precision (i.e., the proportion of true positives among all the instances classified as positive) between 61 and 65% for the conventional neuropsychological assessment of executive functions (Table 5 panel A), while it ranged from 79 to 86% for EXIT 360° (Table 5 panel C) and from 80 to 90% for traditional battery and EXIT 360° together (Table 5 panel B).
Interestingly, all machine learning algorithms showed that the indices from EXIT 360° had a higher capability in predicting PD Group membership compared to traditional neuropsychological tests of executive functioning (Fig. 4).
Usability. The comparison between the PwPD and HC showed the absence of a significant difference in usability score (F (1,78) = .415; p = .521). Indeed, PwPD provided a mean score of 77.3 ± 9.30, while HC showed a mean score of 75.7 ± 12.41. Both scores indicate a satisfactory level of usability, according to the scale's score acceptability ranges (cut off = 68) and adjective ratings (included between "good" and "excellent"). Moreover,   www.nature.com/scientificreports/ Pearson's correlation showed no significant relationship between the total usability score and, respectively, EXIT 360° Total score (p = .711) and EXIT 360° Time score (p = .560).

Discussion
This work aimed to determine whether EXIT 360° could integrate the conventional paper-and-pencil neuropsychological assessment in PD, providing a more ecologically valid evaluation of executive functions. Firstly, we examined the performance of PwPD and HC by comparing the traditional neuropsychological battery with the EXIT 360° to evaluate its efficacy in detecting executive deficits. Correlations between performances on EXIT 360° and paper-and-pencil tests were also investigated. Finally, we looked into the predictive validity of EXIT 360° scores in discriminating PwPD from HC in executive functioning. All subjects were in a relatively wellpreserved clinical state. However, the neuropsychological assessment of executive functions showed differences between patients and HC in two tests (FAB and PMR). Correlation analyses indicated that neuropsychological tests correlate significantly with EXIT 360° scores, supporting a good convergent validity 14    www.nature.com/scientificreports/ 360° can be considered a tool able to detect several components of executive functioning, including cognitive flexibility, inhibition control, sustained and selective attention and processing speed. As regards analyses on EXIT 360°, our main findings revealed significantly different performances between patients and cognitively healthy participants in both EXIT 360° scores. Specifically, PD patients, compared with HC, made more errors in completing the subtasks of EXIT 360° and took longer to conclude the test (i.e., a slower processing speed). These results showed that EXIT 360° is an ecological tool highly sensitive to executive impairment in PD since the mild-to-moderate stage of PD (Hoehn e Yahr scores < 3) when motor symptoms outweigh the cognitive ones. This assumes considerable importance in clinical practice since executive deficits in the early stage of PD are predictive of the conversion to dementia 20,45-47 , with a negative impact on everyday functioning 8,24,25 . Therefore, identifying patients with a potentially higher risk of dementia appears to be a priority to plan an early and individualized cognitive rehabilitation treatment 22,28 .
Interestingly, our results align with a previous study on PwPD that showed the efficacy of a VR-based instrument, VMET, in evaluating executive impairments, which had not been fully acknowledged by traditional neuropsychological evaluations 22 . It is well known that the most crucial issues of the traditional neuropsychological tests are the lack of ecological validity and the ability to measure only one specific component of executive functions without reflecting an accurate and complex picture of a patient's executive status 7,8,48 . For this reason, patients with presumed executive deficits can perform similarly to HC on traditional neuropsychological tests yet encountering difficulties in real-world situations 22 . In this context, the technology 360° may be used to offer a new paradigm in which patients are active participants within an ecological virtual world 1,49 in which it is possible to simulate life-like challenges that reproduce everyday situations. In this framework, EXIT 360° has proved to be an innovative tool for detecting executive dysfunction through a function-led approach that combined experimental control with an engaging real-world background.
In addition, results obtained from ROC curve analyses clearly indicated the potential of EXIT 360° scores (accuracy and completion time) in distinguishing between PwPD and HC in terms of executive functioning. Specifically, a total score of ≤ 11 allows for accurate (AUC = .897) discrimination of PwPD compared to HC with high sensitivity and specificity. The same results appear considering the total time score, where a value > 600 allows for accurately (AUC = .884) discriminating between patients and controls in terms of processing speed. Moreover, ROC curve analyses allowed us to demonstrate the ability of EXIT 360° Total Score and EXIT 360° Time Score to discriminate between HC and PwPD with better overall prediction accuracy, sensitivity, and specificity than traditional paper-and-pencil neuropsychological assessment for executive functioning. Conventional neuropsychological tests, such as FAB, are stronger tools when the cognitive deficit is overt. In line with the previous evidence, EXIT 360° would be a tool more sensible in the early stage of executive impairments since it collected some ecological aspects that impact everyday functioning 22,50,51 .
These findings were also confirmed by the higher diagnostic accuracy in machine learning classification of participants to the clinical or non-clinical conditions (when using indices from EXIT 360°) with respect to those from neuropsychological assessments. These robust findings demonstrated the efficacy of EXIT 360° for detecting impairment of several components of executive functioning at an early clinical stage of PD. Therefore, EXIT www.nature.com/scientificreports/ 360° can be considered an ecological tool highly usable for prompt diagnosis of executive dysfunction and early enrolment of patients in targeted rehabilitation. Interestingly, machine learning analyses have also suggested that integration between neuropsychological tests and EXIT 360° could allow better classification accuracy, with precision ranging between 80 and 90%. This result supported the potentiality of EXIT 360° to integrate the traditional neuropsychological assessment of EFs in PD with a more ecologically valid assessment.
Overall, our results confirm that the 360° technology may play a key role in neuropsychological assessment, in accordance with previous studies 5 . In particular, our findings follow previous evidence on the 360° version of the Picture Interpretation Test (PIT) ability to detect executive dysfunction in active visual perception, typical of PwPD compared to HC. EXIT 360° can be considered an evolution of the PIT 360°, allowing to evaluate several components of executive functioning in an ecological context. The multicomponent aspect appears critical in the clinical evaluation of PwPD since several studies have shown the presence of several executive impairments in PD, such as planning, attention, working memory, set-shifting, dual-task performance, inhibitory control, and decision making, including social-cognition abilities 9,13,52,53 . In addition, EXIT 360° reproduces everyday domestic settings, such as the kitchen, bedrooms, living room, and landing, allowing an evaluation of the executive impairments in the environment most experienced by the subject, with wide implications also in terms of rehabilitation. This feature is peculiar to EXIT 360° since all technological-based tools for evaluating executive functions in PD have involved only a few everyday life scenarios, especially supermarkets but never domestic environments 13 . Finally, in line with a previous study 16 , EXIT 360° obtained good to excellent usability score and the absence of correlation between total usability score and EXIT 360° indexes. Therefore, low performances of participants did not depend on technological usability.
While the current study's findings are promising, some limitations and future perspectives should be considered. Firstly, to fully evaluate the potentiality of EXIT 360° as a new screening tool for executive functions, future studies are needed to assess its test-retest and inter-rater reliability. Additionally, although participants do not need to move in the environment, they must explore the environment by moving their heads; therefore, it cannot be excluded that EXIT 360° could involve other cognitive domains, such as motor representation and programming. Future studies should investigate this aspect by including measures of motor functioning in a regression analysis along with executive functions. Moreover, it will be important to investigate the value of EXIT 360° in detecting executive impairments in other neurological populations known to have executive dysfunctioning, such as Multiple Sclerosis, Mild Cognitive Impairments and Alzheimer's Disease. Finally, it will be of fundamental importance to develop and validate a parallel form of EXIT 360° to make possible a short-term revaluation in a rehabilitation process.
In conclusion, this study offers evidence that a more ecologically valid evaluation of executive functions is more likely to detect subtle executive deficits in PD patients. EXIT 360° captures early executive dysfunctions of PwPD with better accuracy than the traditional neuropsychological assessment. In this context, we think EXIT 360° has a great potentiality in integrating the traditional paper-and-pencil neuropsychological assessment in PD, with a more ecologically valid assessment of executive functions. This innovative 360°-based instrument, easily accessible and clinically usable, can radically transform patients' and clinicians' assessment experience. Firstly, the times for evaluating executive functionality will be drastically reduced since EXIT 360° lasts at most 15 min. In addition, neurologists and neuropsychologists can get ecologically-valid multicomponent evaluations of executive functioning in PD, collecting the real executive status of patients. The ecological assessment will allow tailoring rehabilitation to the everyday subject's needs. As previously said, a timely intervention on executive dysfunction in early-stage non-demented PD could minimize the impact of this significant clinical non-motor symptom, increasing the patient's daily functioning and quality of life 20,21 . Interestingly, as it was designed, EXIT 360° could also be used by a streaming platform that would allow to carry out the remote assessment, overcoming the social distancing limits.